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Description 

METHOD AND APPARATUS FOR CALIBRATING GEOMETRICALLY 
AN OPTICAL COMPUTER INPUT SYSTEM 
Croas-Referenoes to Related Applications 
5 This application is a continuation-in-part in 

application of U.S. Patent Application Serial No. 
07/433,029, filed November 7, 1989, entitled "COMPUTER 
INPUT SYSTEMS AND METHOD OF USING SAME", and of U.S. 
Patent Application Serial No. 07/611,416, filed November 
10 9, 1990, entitled "METHOD AND APPARATUS FOR CALIBRATING 
AN OPTICAL COMPUTER INPUT SYSTEM", each one of said 
applications being incorporated herein by reference as 
fully set forth herein. 
Tgcfrnical Field 

15 The present invention relates in general to a method 

in apparatus for calculating geometrically an optical 
computer input system. It more particularly relates to a 
system for calibrating geometrically for distortion in 
connection with such an optical computer input system is 

20 shown and described in the said patent application. 
Background 

A new optical computer input system is shown and 
described in said parent patent application. Such system 
enable the use to shine a high intensity light onto a 

25 screen bearing a computer generated image to provide 
auxiliary information for the computer. Such an input 
system includes an optical sensing device, such as a 
charged coupled device camera focused on to the screen. 
Thus the system can detect high intensity light images 

30 and discriminate them from the computer generated images, 
to input information interactively into the computer, in 
a convenient manner, even in very low ambient light 
conditions. 

While such a computer input system and method of 
35 using it has proven to be highly satisfactory, it would 
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be desirable to provide for a geometric compensation or 
correction* There are various reasons why such a 
geometric correction is required. Firstly, the screen 
onto which is projected the computer generated image may 
5 not be a perfect rectangle as presented to the sensing 
device or camera. In this regard, the screen may be 
tilted either forward or backward , or from side to side, 
or any combination thereof. Thus, the sensing device or 
camera will not track properly relative to the image 
10 visualized from the screen. 

An even more signif icant problem is the problem of 
"keystoning" which is caused by an overhead projector 
utilized in projecting the computer generated image onto 
the screen. In this regard, the commonly known 
15 keystoning problem produces an image which has a longer 
top edge as compared to its bottom edge. Such keystoning 
problem is well known with overhead projectors, and thus, 
the sensing device or camera will produce a distortion in 
sensing the image projected into the computer. 
20 A third problem is caused by the improper alignment 

of a projection panel on the stage of the overhead 
projector. In this regard, if the panel is not 
accurately aligned in a parallel manner on all sides 
relative to the projector's stage, the resulting image 
25 projected onto the screen will also be askew. 

A fourth problem relates to the project itself not 
being properly aligned relative to the screen. Such is 
commonly the case where the neck portion of the overhead 
projector may be bent slightly due to excessive use or 
30 wear. This causes a result similar to the improper 

registration of the panel on the stage of the projector. 

A still further problem of geometric alignment is 
caused by the camera or sensing device being tilted at an 
angle being tilted upwardly, or downwardly, relative to 
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the plane of the screen. The result is that a distortion 
may occur. 

As a result of both described distortions due to the 
geometry of the screen, projector and panel, as well as 
5 the camera itself, the camera is unable to accurately 
plot the various coordinates visualized from the image 
projected on to the screen. As a result, tracking is not 
able to be perfectly accomplished. Thus, when a light is 
projected onto the screen, the camera may not accurately 
10 know the precise coordinates of the spot of light 

projected onto the screen. As a result, the computer may 
not accurately respond to the position of the light and 
incorrect data can be entered. Thus, erroneous results 
might occur. 

15 Biaaiflaagfl Pff ipygnfrton 

Therefore, the principle object of the present 
invention is to provide a new and improved geometric 
correction arrangement for an optical imaging system. 

Briefly, the above referenced object is realized by 
20 providing a new and improved geometric correction system. 

There is provided in accordance with the present 
invention, a geometric system which includes an 
arrangement for generating geometrically compensated 
relative coordinates for a projected image and for 

25 storing such coordinates. 

Therefore, the system of the present invention 
produces a normalization of the image of the screen to 
provide for the necessary correction. Thus, the 
resulting coordinates stored by the system are 

30 continuously used to ad ast or convert the coordinates 

sensed by the camera. Thus, suitable corrections for the 
distorted are produced. 
Brief Description of Drawings 
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The above mentioned and other objects and features 
of this invention, and the manner of attaining them will 
become apparent, and the invention itself will be best 
understood by reference to the following description of 
5 the embodiment of the invention in conjunction with the 
accompanying drawings, wherein: 

FIG. 1 is a block diagram of the imaging system; 
FIGS. 2 through 11 are diagrammatic views of various 
images or portions of images helpful in understanding the 
10 operation of the present invention; and 

FIGS. 12 through 17 are flow charts of computer 
software for the system of FIG. 1 to illustrate the 
operation of the geometric correction arrangement. 
Beat Mode for Carrying out the invention 
15 Referring now to the drawings and more particularly 

FIG. 1 thereof, there is illustrated a computer input 
system 10 which modifies computer generated images 
appearing on a screen 21, and which is constructed in 
accordance with the present invention. The computer 
20 input system 10 generally includes an image 

projection/detection system or arrangement 11 whose input 
path (cable 17A) is coupled to the output of a video port 
17 of a computer 16. The arrangement 11 comprises a 
liquid crystal panel 13 and a charge coupled device image 
25 sensor 14. The computer 16 is a conventional personal 
computer, such as a model PS/2 personal computer 
manufactured by International Business Machines. The 
computer 16 includes a video monitor 19A and keyboard 
19B. The panel 13 is driven by the computer 16, for 
30 generating live images which are projected by an overhead 
head projector 22 onto the screen 21. 

The computer input system 10 also includes a signal 
processing unit 25 coupled between the output path (cable 
14A) of the image/detection arrangement 11 and the input 
35 serial port 18 of the computer 16 via cable 25A. The 
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computer input system 10 further includes a light wand or 
light generating pointing device 24 , or a laser light 
generating device. 

The projection/detection arrangement 11 detects the 
5 presence of auxiliary light image or spot projected onto 
the viewing surface 21 by the handheld/battery-operated 
light generating device 24, and generates an analog 
electrical signal which is coupled to the signal 
processing unit 25 via cable 14A. The signal processing 

10 unit 25 responds to the analog signal, and converts the 
signal into digital pixel coordinates reference signals 
which identify the relative position of the auxiliary 
light image on screen 21, which are transferred into the 
computer 16 via the output cable 25A. Cable 25A is 

15 connected to the serial input port 18 of the computer 16. 
Computer 16 responds to the pixel coordinates signals and 
can alter its application program which causes the 
computer generated image being projected onto the screen 
21 to be modified. For example, the computer generated 

20 projected image on the viewing area 21 can be modified in 
accordance with the information contained in the 
coordinate references signals. 

In the firmware stored in the signal processor 25 
provides the necessary geometric correction for the image 

25 in accordance with the present invention. As shown in 
FIG. 2, the correction commences by projecting a bright 
rectangular light onto the screen to determine what 
correction is necessary and then record the necessary 
information for converting the coordinates to an adjusted 

30 relative coordinate. As shown on FIG. 2, a true 

rectangular image is indicated at 80. It should be noted 
that each one of the four sides of the image can be 
distorted in a generally rectangular manner. On each 
side of the rectangular image 80 there are two possible 

35 triangular areas of distortion possible. The arrangement 
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of the present invention determines which one of the two 
triangular areas of distortion are present for each side 
of the rectangular image. Once that determination is 
made, a formula for the relative correction is generated 
5 and stored in the signal processor. Thus, there are 
eight possible triangular areas of distortion indicated 
at 81 through 88. 

In FIG. 3, there is shown an example of a grossly 
distorted rectangular image 90 as an example. The first 

10 portion of the process or technique for doing the 

correction, is the actual corners of the projected image 
are shown. The coordinates for the individual corners 
are shown in FIG. 3. The technique for determining the 
corners are similar as shown and described in the parent 

15 applications. 

Once the corner coordinates are generated, a defined 
central coordinate of X , Y is located at the 
intersection of the diagonals of the corners as indicated 
in FIG. 3. Thus, it is important to know for the 

20 purposes of the invention four quadrants of the generally 
rectangular image. In this regard, if the coordinate 
identified on the screen is, for example, in the upper 
left quadrant, then one of the two top triangles must be 
identified, together with one of the two left triangles. 

25 In so doing, the relative coordinates for the identified 
spot of light on the screen can be determined 
accordingly. 

Referring now to FIGS. 12 through 17 , the software 
for the geometrically correct arrangement will now be 

30 considered in greater detail. Once a rectangular image 
is projected onto the screen, such as the screen 21 of 
FIG. 1, the software in the signal processor 25 
determines the coordinates of the four comers of the 
image appearing on the screen, as indicated at box 121 of 

35 FIG. 12. In this regard, the image sensor 14 (FIG. 1) 
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determines the interface between the bright image and the 
remaining portion of the screen, in a similar manner as 
described in the parent patent applications. 

As indicated in box 122 in FIG. 12, and as shown in 
5 FIG. 3, the geometric center of the screen is determined 
and defined at the intersection of diagonal *** lines 
extending through the corners. And thereafter, as 
indicated in box 123, particular coordinates of a spot of 
light projected onto the screen by the handheld light 

10 device 24 is determined. In this regard, a position X of 
the spot of light is determined to be either left of 
center or not. If it is determined to be left of center, 
then the software will analyze whether the left edge is 
in perfect vertical alignment or whether it is distorted 

15 according to either one or the other triangular areas of 
distortion 81 or 82, as indicated more clearly in FIGS. 4 
and 5. As indicated in box 124, a determination is made 
as to whether or not the left edge is a perfectly aligned 
vertical edge. In this regard, the X coordinate of the 

20 top, left corner is compared with the X coordinate 

bottom, left corner. If they are equal, then the edge is 
determined to be perfectly oriented in a vertical 
disposition, and thus correction is not required. If 
such occurs, box 131 of FIG. 13 is entered. 

25 Assuming that a left margin of the image is required 

to be corrected, the decision box 125 is entered as shown 
in FIG. 12 to determine whether the X coordinate of the 
top left corner is greater than the bottom left corner. 
This determination is made to learn whether there is 

30 rectangular distortion area 81 (FIG. 4) , or a triangular 
area of distortion shown at 82 of FIG. 5. The triangular 
area 81 is in the shape of a right of a triangle having 
its based aligned with the bottom edge of the rectangular 
image. The rectangular area 82 is inverted from the area 
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81, and has its base coextensive with the top edge of the 
rectangular image area. 

At the decision box 125,. if the X coordinate at the 
top left corner is greater than the X coordinate at the 
5 bottom left corner then it is determined that a 
triangular image area 82 is present. In such a 
situation, the box 126 is entered and the left edge X 
coordinate is calculated in the box 126. Thereafter, at 
the box 132, 133 the absolute value of the X coordinate 

10 of the spot is calculated. This value of the X 

coordinate is then used for sending to the computer 16, 
instead of sending of the value of the X coordinate of 
the spot of light as detected by the sensor 14. In this 
regard, the formula shown in the boxes 132 and 133 are 

15 used to scale the X coordinate of the spot of light on 
the screen 21 to correct geometrically for any 
distortions. It should be understood that this X 
coordinate correction is calculated on the slide for each 
spot of light detected by the image sensor 14. 

20 Thus, it is to be understood that the initial 

storing of the X and Y coordinates of the four corners 
and the defined center is the only information that need 
be saved during initialization process. Thereafter, the 
absolute values of the X and Y coordinates of the spot of 

25 light are calculated and supplied to the host computer 16 
on the fly. 

In order to determine the other left edge triangular 
area of distortion indicated at 81 at FIGS. 2 and 4, the 
decision box 125 of FIG. 12 will determine that the top 

30 left X coordinate is not greater than the bottom left X 
coordinate, so that the box 126 is entered to calculate 
the left edge X coordinate. After so calculating, the 
boxes 132 and 133 are entered to determine a different X 
ABSOLUTE value based upon the triangular area 81 of 

35 distortion. 
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Referring again to box 123 to where the X coordinate 
position of the light spot is determined to be left of 
center, if it is not left of center then the decision box 
141 of FIG. 14 is entered. A determination will then be 
5 made as to whether the right edge of the rectangular 
image area is perfectly vertical, or whether one of the 
two triangular areas 83 and 84 at the right edge of the 
image area occurs. This operation is similar to the 
operation of distinguishing the top triangular areas 81 
10 and 82. 

At the decision box 141, there is determination made 
as to whether the top right X coordinate is equal to the 
bottom right X coordinate. If they are equal then box 
145 is entered directly to determine if the right edge X 
15 coordinate is equal to the X coordinate of the top right 
edge. 

However, if they are not equal, then the box 142 is 
entered to determine whether or not the top right X 
coordinate is greater than the bottom right X coordinate. 

20 If it is, then the right edge X coordinate is calculated 
by the formula shown in box 143. Thereafter, as 
indicated by box 151 and 152, the absolute value of the X 
coordinate is then calculated knowing the right edge X 
coordinate and the X coordinate of the light spot and the 

25 X coordinate of the defined center. This absolute value 
of X is then used and supplied to the host computer 16. 

Referring again to the decision box 142, if the top 
right X coordinate is not greater than the bottom right X 
coordinate, then the triangular area 83 (FIG. 2) is 

30 identified and the box 144 is entered to calculate the X 
coordinate of the right edge, thereafter, calculations 
are made at boxes 151 and 152 calculating the absolute 
value of the X coordinate. 

Once the absolute value of the X coordinate is 

35 determined from either one or the two triangular areas 83 
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or 84 , absolute value of the Y coordinate is then 
calculated. 

In order to calculate the relative value of the Y 
coordinate of the light spot, the quadrant of the light 
5 spot is first determined at box 153. In this regard, at 
box 153 a determination is made as to whether or not the 
Y position of the spot is above the defined center. If 
it is, then the decision box 154 is entered to determine 
whether the top edge is either horizontal or whether 

10 there are either one or two triangular areas 85 or 86 as 
shown in FIG. 2. 

If a determination is made at box 154, if the top 
left Y coordinate is not equal to the right Y coordinate, 
then at the decision box 155, a determination is made as 

15 to whether or not the top left Y coordinate is less than 
the top right Y coordinate. This decision will identify 
which one of the two top triangular areas 85 or 86 is 
present. - If the decision is positive, then the 
triangular area 86 is present. If the decision is 

20 positive, then the triangular area 86 is identified and 
the calculation shown in box 156 is made to determine the 
top edge Y coordinate. Thereafter, as shown in box 163 
(FIG. 16A) , and box 164 of FIG. 16A, the absolute value 
of Y is then determined for supplying it to the host 

25 computer 16. This value of the absolute value of Y is a 
scaled value of the Y coordinate of the spot. 

It the top triangular area 85 is present, a similar 
calculation is made at box 161 for the top edge of the Y 
coordinate of the triangular area 85. Thereafter, the 

30 absolute value of Y is then calculated boxes 163 and 164. 
Once the absolute value of the Y coordinate is 
calculated, a box 176 in FIG. 17 is entered to prepare 
and transmit to the host computer once both absolute 
value of the X coordinate and the absolute value of the Y 

35 coordinates have been calculated. It should be noted 
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that once these absolute values have been transmitted, 
the routine loops back to the initial box 123 to repeat 
the cycle of operation for the next light spot detected. 
Referring again to box 153 of FIG. 15 , if it is 
5 determined that the Y position of the spot is not above 
the center, then the decision box 165 of FIG. 16 is 
entered. A determination is then as to whether or not 
the bottom left Y coordinate is equal to the bottom right 

Y coordinate. The purpose of this determination is to 

10 decide whether the bottom edge of the rectangular viewing 
area is either horizontal, or at a triangular position as 
indicated at either 87 or 88 in FIG. 2. If it is 
determined that the bottom edge is a true horizontal 
line, then the boxes 173 through 175 are entered to 

15 calculate the value of the absolute value of Y as 

previously explained. On the other hand, if they are not 
equal, then the decision made in box 165 determines 
whether or not the bottom left Y coordinate is greater 
than the bottom right Y coordinate. If it is, then the 

20 box 171 is entered to perform a calculation to determine 
the bottom edge value of the Y coordinate. This 
calculation is based on the triangular area 88, since the 
bottom left Y coordinate is greater them the bottom right 

Y coordinate. If the reverse is true, then the 

25 calculation is made at box 172 based on the triangular 

area 87 to determine the bottom edge Y coordinate. 

Thereafter, the absolute value of the Y coordinate is 

calculated as previously described. 

The following is a series of the equations used to 
30 perform the various calculations as illustrated in the 

flow charts to scale the X and Y coordinates of the 

light spot. 

1) FIRST DETERMINE X, Y POSITIONS OF EACH 4 CORNERS 

2) NEXT, TO FIND CENTER OF SCREEN, FIND INTERSECTION OF 
35 DIAGONAL LINES: 
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REFERRING TO FIG. 3, CALCULATE: 

slope - _ y v _v 



POINT SLOPE FORM OF EQUATION OF LINE 

v-Ybr = (x-x ER ) y-Y BL = a^U-x^) 

y-Faj = mzpX-mznXgz y~ Y BL • x ~ m RL X BL 

- lymjgX = Y^-M^XsJ 

(m^-m^) X c = Ybr-^I^BR- Y BL + m RL X BL 



(2SSm I£ )X BR +2 56 Y^ * (~256in m ) 
= (256/n^) + (-256m m ) 

SUBSTITUTING INTO FIRST EQUATION 
Y c m MlrXc-MlrXbr + Y BR 

V c = ^LR^ - m LR X BR +Y BR 

3) DETERMINE IN WHICH QUADRANT, THE SPOT APPEARS BY 

APPLYING VERTICAL & HORIZONTAL CENTER LINES THROUGH 
THE CALCULATED CENTER. 
4) COMPUTE THE ADJUSTED EDGE FOR BOTH X & Y IN THE 
FOUND QUADRANT. 



WO 92/15084 



PCT/US92/00876 



TANa = 



13 

SIDE OPP 



SIDE ADJ 

SIDEOPP (ADJUSTMENT) = (-DJSTAATCZ? SIDE ADJ) (TANa) 
REFERRING TO FIG. 4, FOR A LEFT EDGE OF 

wfcere (IW-^ 35 DISTANCE SIDE OPP and where 

X RBLATivB= X RAir X Bom NOTE I IF 
IS NEGATIVE THEN POSITION IS OUTSIDE OF SCREEN 

X—U095) 



X ABSOLOTB " v _v 



REFERRING TO FIG. 5, FOR A LEFT EDGE OF 



(NOTE: IF JW IS NEGATIVE THEN POSITION IS OUTSIDE OF SCRE1 



X^ ; 4095 



REFERRING TO FIG. 6, FOR THE RIGHT EDGE OF 
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(NOTE: IF X RSS > THEN OUTSIDE OF SCREEN) 

Xrbl ' 4095 
= 4096 + HP TiT" 

REFERRING TO FIG. 7, FOR A RIGHT EDGE OF 

(NOTE'. IF Xjow > Xmggr THEN OUTSIDE OF SCREEN) 

Bright a c 



REFERRING TO FIG. 8, FOR A TOP EDGE OF 



(NOTE: IF < Y BOHB THEN OUTSIDE OF SCREEN) 

Yrbl ; 4095 



REFERRING TO FIG. 9, FOR A TOP EDGE OF 
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(NOTE: IF Y^ < Yg^ THEN OUTSIDE OF SCREEN) 



_ • 4095 



Ye-Y, 



ABS 

REFERRING TO FIG. 10, FOR A BOTTOM EDGE OF 

Y BOTTOM = Y BL " ^ X RAlT X Bl) 



( Ybl-Ybx } 



(NOTE: IF Y^ > Y BOT7txf THEN OUTSIDE OF SCREEN) 



■'■BOTTOM z c 



REFERRING TO FIG. 11, FOR A BOTTOM EDGE OF 



WO 92/15084 



PCT/US91/00876 



16 

(NOTE: IF Y m > Ybottom THEN OUTSIDE OF SCREEN) 

iW 4095 
Y m = 4096 + — 

1 BOTFOH z c 

While particular embodiments of the present 
invention have been disclosed, it is to be understood 
that various different modifications are possible and are 
contemplated within the true spirit and scope of the 
5 appended claims. There is no intention, therefore, of 
limitations to the exact abstract or disclosure herein 
presented. 
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CLAIMS 

What is claimed is: 

1. An apparatus for correcting image distortion of 
a displayed rectangularly shaped image, characterized by 

5 a storage device for storing determined coordinate 

locations of the displayed image and a processing system 
for calculating corrected coordinate values relative to 
the stored determined coordinate locations to 
substantially eliminate coordinate errors resulting from 
10 image distortion in the displayed image. 

2. A method for correcting image distortion of a 
displayed rectangularly shaped image, characterized by 
storing determined coordinate locations of the displayed 
image and calculating corrected coordinate values 

15 relative to the stored determined coordinate locations to 
substantially eliminate coordinate errors resulting from 
image distortion in the displayed image. 

3. An apparatus according to claim 1 characterized 
in that said processing system includes a projection 

20 device for projecting an x,y coordinate image of light to 
any given location within the area of the displayed image 
bounded by its four respective edge portions; an image 
detecting device for determining the quadrant of said x,y 
coordinate image disposed in the displayed image; and a 

25 microprocessor for calculating an adjusted edge for both 
the x and y stored determined coordinate locations in the 
quadrant of said x,y coordinate image. 

4. An apparatus according to claim 3, further 
characterized by an algorithm for determining the corner 

30 x,y coordinate values for each respective corner of the 
displayed image; and another algorithm for determining 
the x,y coordinate value for a defined center within the 
displayed image area, wherein said image area is bounded 
by the four respective edges of the displayed image. 



WO 92/15084 



PCT/US92/00876 



18 

5. An apparatus according to claim 4 characterized 
in that the algorithm for determining the x,y coordinate 
value for a defined center includes a point slope 
determining algorithm. 
5 6. An apparatus according to claim 5 characterized 

in that said point slope determining algorithm solves an 
equation: 

slope m TJ? = y ™~ y s* = y **~ y ™ 

LR xr y y — V 

A-TtT^ER BR TL 

wherein y TL is indicative of a determined Y to 
left coordinate value: 
10 wherein Y^ is indicative of a determined y 

bottom right coordinate value; 

wherein is indicative of a determined X top 
left coordinate value; and 

wherein Xg R is indicative of a determined Y 
15 bottom right coordinate value. 

7. An apparatus according to claim 6 characterized 
in that said point slope determining algorithm further 
solves a pair of equations: 



8. An apparatus according to claim 1 further 
20 characterized by an algorithm for determining the x,y 
coordinate locations of each respective corner of the 
displayed image; and another algorithm for determining a 
defined center coordinate location for the displayed 
image. 

25 9. An apparatus according to claim 3 characterized 

in that said microprocessor for calculating an adjusted 
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edge includes an absolute valve algorithm for calculating 
an absolute y value and an absolute x value. 

10. An apparatus according to claim 9 characterized 
in that said absolute y value is: 



4096 y, 



relative 



absolute xr _ \r 

* bottom edge x c 

5 11. An apparatus according to claim 10 

characterized in that said Y pelative is: 

^relative = Y 3POT " Y c 

12. An apparatus according to claim 11 
characterized in that said y spot is indicative of y 
coordinate value of the projected x,y coordinate image of 

10 light. 

13. An apparatus according to claim 9 characterized 
in that said absolute x value is: 

Y _ x relative » 4095 

^absolute " \r —Y 

A C LEFT EDGE 

14. An apparatus according to claim 13 
characterized in that said X^^,^ is: 

^relative ~ X SPOT ~% X eft edge 

15 15. An apparatus according to claim 14 

characterized in that said Xspot is indicative of the x 

coordinate image of light. 

16. An apparatus according to claim 15 

characterized in that said X left edge is X TL . 
20 17. An apparatus according to claim 16 

characterized in that said X leftedge is X TL . 
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